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Despite the increasing number of programs for youth now operating (such as 
school-to-work, mentoring, employment, tutoring, and recreational), very little is known 
about their quality or impact. Some evaluations, anecdotal evidence, and the 
impressions of the participants do, however, suggest that the programs are successful. 
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Everyone believes that only good can come from them, that at the worst they will not 
accomplish all that they could, but that youth will be better for the experience, even if it 
has not markedly affected their lives. But if these programs are to achieve their highest 
potential, they need to be nurtured, managed well, and evaluated. Yet, program 
officials, policy makers, and the community have only limited information about how 
best to implement these programs and about which program features are the most 
essential and beneficial to students. Neither do they know how much the programs help 
students develop the career and academic orientation necessary to stay and succeed in 
school, go on to postsecondary and higher education, and be prepared for economically 
sound and personally gratifying work. 

Given the choice to spend money to evaluate a youth program or to provide more 
services, most program administrators would not choose the evaluation. They would 
rather use the funds to reach more youth, recruit and train more staff, or provide more 
experiences for the youth. This is unfortunate. Ongoing assessment, feedback, and 
evaluation are as important to the health of a program as are its design and 
management. Ideally, they should be made part of the design and operation of a 
program at the outset, not as an afterthought under pressure from a funding agency or 
the community or to showcase the "success" of a program. The findings of an 
evaluation should be reviewed periodically and be the basis for modifying the program. 
This can be done through process and impact evaluations of the program. 

This digest examines features of both process and impact evaluations in order to help 
officials take simple actions to assess the programs for which they are responsible. With 
proper planning, evaluation can be part of a program without having to compete for 
attention and resources with the services the programs are trying to provide. 

STRATEGIES FOR PROGRAM EVALUATION 



Process evaluations provide information about the operation of the components of a 
program; impact evaluations explain whether they are yielding the intended outcomes. 
Process evaluations consider the components of a program; impact evaluation, the 
outcomes. These types of evaluation serve different purposes, but they necessarily 
depend on each other. The impact of a program can only be adequately determined by 
ascertaining whether and how well it has been implemented. In turn, learning the 
outcomes of a program sheds light on its operation, particularly if the intended results 
were not attained. And both types of evaluation can be used to refine the design and 
management of the program to better achieve the desired outcomes. 



PROCESS EVALUATION 

A process evaluation examines the design, implementation, and context of a program, 
and its data collection, analysis, and reporting activities. 
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ADHERENCE TO DESIGN. In a process evaluation the program staff must first 
determine whether a program design has been altered through its implementation. In 
truth, no design can be faithfully followed because local conditions over a period of time 
affect how it will be implemented. Some changes have little or no effect on the program 
model, and leave it basically intact. Some minor changes may even be beneficial. But 
some adaptations or changes can so severely alter a design during implementation that 
it no longer resembles the original design. Sometimes the program design itself can 
throw the program off course, and, thus, necessitate changes. For example, some 
features of the design may be too complicated to be easily implemented; or the goals of 
the program may be too ambitious or vague, possibly leaving the staff to misinterpret 
the intent. Sometimes the staff members, unprepared or not free to implement the 
design of a program, have to improvise the implementation. 

Any evaluation of the adherence to the program design requires determining the core 
components of the model, the extent to which each component is actually implemented, 
the degree to which the clients and service providers participate in all or the appropriate 
aspects of the program, and the consistency of any changes in the design with the 
purposes and objectives of the program. 

IMPLEMENTATION. Programs can be implemented either to increase the probability 
that the design will work or to reduce its chance for success. Data for documenting the 
implementation of a program design should indicate the role and extent of all the 
program participants, the training of the staff, and the time and resources allotted to the 
program. It is critical that all aspects of the implementation of the program be 
documented, from the point the model is put into practice, times and nature of any 
changes, to the completion of the program. 

CONTEXT. Program officials and others sometimes do not realize that programs 
operate within a social and political context, which affect the outcomes. First, the 
characteristics of the program participants and the school and community can affect the 
implementation and outcomes of a program. For example, some youth are more 
amenable to a particular kind of program than others, and some parents are more 
supportive than others. Second, the preparation of the staff, the adequacy of financial 
resources, conflicting or competitive programs, and the social and political climate all 
affect a program. Further, program staff often must meet demands to implement several 
new programs at the same time. Finally, lack of funds may cause premature or 
compromised implementation of a program. In examining the effect of the context of the 
program on its implementation, however, it is important to distinguish between 
significant and negligible influences. 

DATA COLLECTION. Data should be collected at key points throughout the life of a 
program. At the outset program officials and staff should design simple recordkeeping 
procedures and follow them regularly. Similarly, they should create a management 
information system to document the number and types of participants in the program, 
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and track their retention and completion rates and their experiences in the program, 
including any changes in their use of services over time. These data can be easily 
obtained from interviews, focus groups, observations, journals and logs, and 
administrative records. 

ANALYSIS AND REPORTING. Program officials need to establish periodic benchmarks 
and guidelines for collecting evaluation information and examining the results. Without 
these benchmarks the collected data are meaningless for analyzing problems and 
altering the course of a program. Ideally, this process should start by creating a 
summary of the development of the program and its initial implementation. Program 
administrators need to examine the critical issues governing the implementation of the 
program, like adherence to the program model (both ideal and necessary features) and 
the types of implementation problems encountered, including their causes and 
solutions. Reflecting on the program as a whole is critical because it allows program 
staff to find more inclusive solutions rather than making incremental decisions as 
problems arise, or having to defuse crises. It also allows them to intervene early before 
problems become endemic and difficult to solve. 

Again, if time and resources allow, it is important to compare the experiences of the 
program with those of similar programs. The comparison is a useful way to find 
solutions to problems common to all the programs. It also helps to determine whether 
problems are a function of the model itself or whether the problems are local to the 
particular school or program setting. 

In short, a process evaluation is a necessary management tool, which should be used 
more than once. It can be used to determine whether a program is being properly 
implemented or is drifting from its original intent, a hazard in any long term activity. 



IMPACT EVALUATION 

Although it is essential for program planners and officials to understand how well a 
program has been implemented, it is also important to demonstrate whether, how, and 
to what degree the program has affected the participants. The simplest evaluation 
question is: Did the program meet its objectives and yield the appropriate outcomes? 
Such an evaluation is critical to determine whether the program should continue in its 
current form. If it can be shown that the program has been effective, it will be easier to 
receive continuing financial, administrative, and community support. But to obtain the 
data to support claims about the effectiveness of a program, program officials must 
devise an appropriate evaluation design and administer it properly. 

OUTCOMES AND THEIR MEASUREMENT. Ultimately, the programs need to improve 
the lives of the participating youth, whether by reducing their risky behavior, improving 
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their academic achievement and progress, or helping them obtain and retain career jobs 
in growth industries. Measures of these outcomes must be tailored to the specific 
activities of the program or the program component being evaluated. Measures include 
the following: pre-program and post-program student grades, performance on 
standardized achievement tests, and attendance and promotion rate; students' 
self-reported confidence in their abilities, knowledge, skills, and success in school and 
later in life; students' self-reported and tested knowledge about education, jobs, careers, 
and the world of work; and teacher evaluations of students' academic performance and 
behavior. In addition, the impact of the program on students' subsequent academic 
performance and other behaviors can be determined by an examination of their records. 

COMPARISONS. The significance of the outcomes for the participants and the 
contribution of the program model to these gains can only be determined by a 
comparison. This can be done in two ways. The first is a within-program comparison, 
the second is a comparison with groups that did not participate in the program. 

The first comparison determines the degree of the program impact on selected 
outcomes among participating youth with different characteristics. The second type of 
evaluation determines whether the program or other factors are responsible for the 
outcomes. This comparison also reveals whether differences in the outcomes are 
attributable to differences in the program participants instead of in the program itself. 
Clearly, demographic differences in the groups being compared: age, gender, and 
socioeconomic status, and differences in their prior academic record: attendance, 
grade-point average, accumulated credits, and history of disciplinary referral, influence 
program outcomes. It is very difficult to determine whether differences in program 
outcomes can be attributed to the impact of the program when the youth being 
compared differ from one another too much. 

Ideally, an evaluation should compare groups identical to each other, but this is not 
usually possible. A larger number of youth than usual would have to ask or be recruited 
to participate in a program in order to have enough students to make the comparison. 
Then some of the youth would have to be assigned to the program, some to an 
alternative, and some to no program at all. Such a scenario is unlikely to happen 
because program officials would have to withhold the program from some youth or a 
lottery system would have to be instituted to determine who can participate the program. 

It is possible, however, to find alternative ways of comparing youth in the program with 
others. Program planners can compare youth in the program with profiles of the 
academic and other characteristics and performance of a representative sample of 
students that are contained in district, state, and national survey data. These data are 
readily available, often in the popular media. 

DATA COLLECTION. Program officials need to plan ahead and obtain data at the 
outset of a program, if they are to evaluate its impact. Enrollment applications, 
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placement information, and initial interviews contain information about students' 
attitudes and knowledge before they enter the program. During the operation of the 
program a management information system can document students' participation, 
progress, and accomplishments. To obtain additional information, the youth can also be 
observed in various activities; they can be interviewed individually, in pairs, and in 
groups; and they can keep individual logs. Finally, the program staff can keep extensive 
case histories on a small sample of participants. 

Interviews, logs, and observations provide descriptive information, but programs also 
need standardized information about the participants' attitudes, knowledge, plans, and 
achievements. This can best be collected from questionnaires and other instruments, 
and from school records. Program officials need not develop these questionnaires. 
Nationally developed instruments that contain items relevant to many kinds of programs 
are readily available. Follow-up questionnaires at six months, one year, and two years, 
using the same survey instruments, are also useful for demonstrating the sustaining 
effects of the program and the participants' progress. 

ANALYSIS AND REPORTING. At a minimum program planners need to conduct two 
kinds of analysis. One kind examines whether the program model produced the 
anticipated outcomes. Another kind of analysis investigates how much the magnitude 
and intensity of the intervention contributed to the outcomes. A third demonstrates the 
differential impact of different kinds of interventions. And a fourth kind of analysis could 
present the program outcomes for subgroups of program participants, like girls. 

Analysis of the data about program outcomes is not enough, however. In presenting the 
results of the analysis the staff needs to consider the possible explanations of the 
outcomes (or lack of outcomes). In the real world of youth programs, staff and youth 
drop out of the program, conditions require that the design of the program be altered, 
and local funds and other support come and go. Moreover, the results should be 
reported and disseminated in a form useful to other program planners. Usually the 
public report need only contain a synopsis of the program characteristics and findings. 
Those needing more specific or other information can pursue their request individually. 

MANAGEMENT OF THE EVALUATION. Although an evaluation is essential to carrying 
out a successful program, it should never be a burden for the staff or participants. It 
should be managed to provide the knowledge needed to conduct the program 
successfully and tailored to fit into the program management. Choosing an outsider to 
conduct the evaluation is not always necessary, although an outsider can provide the 
unbiased point of view necessary to demonstrate the effectiveness of the program to a 
wider audience than can the program planners and staff. Assigning the responsibility for 
the evaluation to the program staff, however, has several advantages. It makes the 
evaluation part of the program management rather than a separate activity; and it 
encourages the staff to use the evaluation process to solve any design, implementation, 
or operational problems. 
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THE REALITY OF ASSESSING PROGRAM 
EFFECTIVENESS 



Most youth programs are small, local, service-driven, and improvised, and these are not 
the best conditions to carry out a program or conduct an evaluation. Nevertheless, 
program staff can take some simple actions to obtain data about the effectiveness of the 
program that will help them improve the program, report the results to administrators 
and funders, and inform other professionals and the public about the quality and 
performance of the program. 

First, at the outset, program planners and staff must commit to determining the 
effectiveness of the program and its impact on participants, and assign staff and time for 
the evaluation. They must include evaluation at every point of the implementation of the 
program-recruiting and selecting the youth, determining and carrying out program 
activities, staffing, and so on. This is no small undertaking, but without it there can be no 
assessment of the program because planners and staff will have no design to evaluate 
and no data to use to interpret the impact of the program. 

Second, to understand the impact of a program, planners must know what the 
"program" is. Youth programs are particularly vulnerable to poorly-articulated goals and 
undefined activities because the mystique of youth programs encourages everyone to 
believe that because only good can come from them nothing needs to be planned. Too 
often both the plans and goals for the programs are global and never specified in ways 
that can be translated into discrete activities whose impact can be evaluated. 

Third, program staff must identify data sources and begin to collect information on the 
implementation of the program and on the characteristics of the participants, before and 
after the program. This is not difficult to do, despite the fears of many program staff, 
because they already have some of the data from student records, and they can collect 
the rest from interviews, observations, logs, and already existing questionnaires 
adapted for local use. It is important, however, to be clear about why the data are being 
collected: for example, to evaluate the effectiveness of a strategy, or to determine 
whether the program has the right components. 

Fourth, these data must be analyzed and interpreted. The analysis, again, can be very 
simple, but still be effective and useful. Quantitative data show absolute changes in 
achievement test scores, attendance patterns, referrals for discipline problems, or 
knowledge about behaviors that avoid risk. Qualitative data indicate relative changes in 
the behavior of program participants. Qualitative data, usually culled from interviews 
and observations, are more difficult to analyze because the program staff must come up 
with checklists and coding schemes to ascertain whether the participants have changed 
as a result of the program, but this, too, can be done if the staff determines the 
desirable program outcomes at the outset. The findings of the evaluation must then be 
interpreted, for data are valueless without an explanation of their meaning. The staff 
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likely will have rich and complex explanations for the findings, based on members' 
experience with all aspects of program. Again, taking the time to interpret the data 
thoughtfully is critical, particularly where there are no discernible outcomes for the 
participants or where the program was never properly implemented. 

Finally, after many evaluations of youth programs, researchers are now realizing that no 
matter how good the policy for developing the program, or its design, some programs 
achieve excellent results, some have no outcomes, and some even have a negative 
impact on the participants. This is true even when the clients and program services are 
similar. It is possible, of course, that data about the program's effectiveness were not 
collected well, or that the findings were not interpreted properly, but it also may be that 
the programs were not managed effectively. 

Youth programs, particularly, often lack stable and mature management structures 
because they are add-ons or afterthoughts. Very often these programs do not have the 
clearly articulated mission and identity that determines the kind of services they provide 
and informs the potential participants and the public about what they can expect from 
the program. Having a mission is crucial, though, to the successful targeting and 
recruitment of both the participants and the right staff to manage the program. Those 
interested in understanding the effectiveness or performance of a youth program need 
to understand that if it is not well-managed, the intervention will not be effective, or 
possibly even be implemented. The only result that the evaluation or performance 
analysis shows, then, is that the program never really happened, despite appearances. 

BIBLIOGRAPHIC NOTE 



The discussion of process and impact evaluations is a summary of material appearing 
in Evaluating School-to-Work Transition (1995), by Margaret Terry Orr, available from 
the National Institute for Work and Learning, Academy for Educational Development, 
Washington, D.C. The discussion of the management of youth programs briefly 
summarizes the discussion in Managing Youth Development Programs for At-Risk 
Students: Lessons from Research and Practical Experience (1992), by Andrew Hahn, 
available from the ERIC Clearinghouse on Urban Education, Box 40, Teachers College, 
New York, NY. 

This Digest was developed by the ERIC Clearinghouse on Urban Education with 
funding from the Office of Educational Research and Improvement, U.S. Department of 
Education, under contract no. RR93002016. The opinions in this Digest do not 
necessarily reflect the position or policies of OERI or the Department of Education. 
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